1. Technical Field
This disclosure generally relates to the field of motion detection techniques in video sequences and, more particularly, to motion detection for temporal mosquito noise reduction.
2. Description of the Related Art
In digital imaging, mosquito noise is one of the commonly seen coding artifacts mainly in smoothly textured regions around high contrast edges as temporal fluctuations of luminance and chrominance levels. It is a form of edge busyness resembling a mosquito flying around a subject. Conventional spatial noise reduction systems could partially reduce mosquito noise, but blurring effect may also be generated as a side effect. Moreover, conventional spatial noise reduction systems also lack of the capability of reducing temporal fluctuation.
Random noise, as another form of noise, exists in captured and transmitted video. It is called random noise as this kind of noise is typically distributed over the images randomly and tends to make the images slightly soft and blurry. On close inspection, one may see tiny specks all over the images as a result of random noise. Random noise may be corrected by a temporal noise reduction system in addition to the spatial noise reduction system for preserving maximum details on stationary scene.
A temporal noise reduction system is a system that performs noise reduction by using information of a pixel at the same coordinates in two consecutive frames. If an image is stationary, the temporal noise reduction system typically shows an excellent noise removal effect. If, however, the image is moving, the temporal noise reduction system tends to deteriorate image quality, resulting in tail artifact or blurring effect. Thus, the strength of the temporal noise reduction system should be accurately controlled, or adjusted, by a motion detection system which indicates the true motion value of a pixel. In noisy images, more sophisticated approach is needed to distinguish between the noise and the true motion, such that the noise may be effectively reduced by the noise reduction system while the details of the images are preserved and no tail artifact is introduced. For efficient mosquito noise reduction, the design challenge of a robust motion detector is the successful handling of strong mosquito noise, especially at areas near strong edges. Such noise might be falsely detected as motion by conventional motion detectors due to its high magnitude, sometimes even higher than that of the true motion pixels.
To date, there have been a number of efforts on motion detection or classification in the context of temporal noise reduction.
For instance, in U.S. Patent Application Publication No. US2006/0158550 A1 filed by Zhou et al., a motion detector for the application of de-interlacing is proposed. This motion detector first thresholds the low-pass filtered frame difference of a pixel. If the pixel or one of its two adjacent pixels delayed by one field has the frame difference larger than the threshold, then the pixel is detected as a motion pixel. The binary decision is then low-pass filtered to give the coefficient for controlling the switch between the temporal and spatial filter. However, this coefficient does not reflect the motion difference value. If there is a region of high motion difference while there is another of low motion difference but classified as motion, both regions will be calculated to have the same coefficient; and thus the same amount of temporal and spatial filtering will be applied. Moreover, the low-pass filter before the pixel classification might destroy the high-frequency edges and details in the difference image and thus make the pixels being misclassified and blurred by the temporal processing.
In S. Skoneczny, “Image processing for old movies by filters with motion detection”, International Journal of Applied Mathematics and Computer Science, vol. 15, No. 4, pp. 481-491, 2005, the author proposed a motion/non-motion pixel classification method by thresholding both the forward and backward differences. If two adjacent pixels have both their forward and backward differences above the respective threshold, the center pixel is classified as a motion pixel. This motion classification system, however, requires processing of three frames, which may involve expensive computation and implementation.
In M. Hensel et al., “Motion and noise detection for adaptive spatio-temporal filtering of medical X-ray image sequences”, Proceedings MIUA, July 2005, the authors proposed a motion and noise detection method for controlling the strength of the spatial and temporal filters to reduce noise, and specifically system noise. The motion and noise differentiation is achieved by morphologically processing the positive and negative pixel values of different images independently. The independent processing of positive and negative pixel difference improves the motion and noise detection. However, the morphological operation requires expensive computation like opening and closing, and thus is not suitable for real-time video processing but off-line image processing.
In S. Delcorso et al., “MNR: A novel approach to correct MPEG temporal distortions”, IEEE Transactions on Consumer Electronics, vol. 49, Issue 1, pp. 229-236, February 2003, a binary motion/non-motion pixel classification by thresholding the low-pass filtered difference image is proposed. Although the low-pass filter is expected to improve the noise robustness of the classification to some extent, it is nevertheless at the risk of destroying the edges or details in the difference image, and thus might result in incorrect classification and blurring.
In International Patent Application Publication No. WO 2006/010276 A1 filed by Dinh et al., a comprehensive 3D post processing method and system is proposed for mosquito noise reduction. The system includes a block localizer, a noise power estimator, a blocking artifact reducer, a spatial noise reducer, a temporal noise reducer, and a detail enhancer. The temporal noise reducer includes a motion detector for minimizing motion blur artifact. The motion detector is adaptive to noise by subtracting the estimated spatial noise variance from the time difference to represent motion. The motion is compared to a threshold value related to the noise variance at the current pixel and a 3×3 window for hard and soft no-motion decision. The motion, the noise variance, and the no-motion decision together yield a final filter coefficient to be sent to the temporal filter. This motion detector, however, is not fully automatic because it requires a noise power estimator which depends on the user correction level. In addition, it requires expensive computation like image segmentation for noise power estimation, in which edge detection is performed for image segmentation. Nonetheless, only strong edges are detected due to a low pass filter before the detection. As a result, blurring effect may be found on soft edges.
Most of the prior art attempted to improve the robustness of the motion detector against noise, but often at the cost of sacrificing small edges and details. Most of the prior art suffer from the absence of edge/texture analysis and protection and, thus, may result in blurring while reducing noise. Although prior art WO 2006/010276 A1 has additional consideration for mosquito noise, it requires user correction and expensive computation such as segmentation for motion detection.